ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO

Image-Text matching (ITM) is a common task for evaluating the quality of Vision and Language (VL) models. However, existing ITM benchmarks have significant limitation. They many missing correspondences, originating from data construction process itself. For example, caption only matched with one image although can be other similar images vice versa. To correct massive false negatives, we construct Extended COCO Validation (ECCV) Caption dataset by supplying associations machine human annotators. We employ five state-of-the-art models diverse properties our annotation process. Our provides $$\times $$ 3.6 positive image-to-caption 8.5 caption-to-image compared to original MS-COCO. also propose use an informative ranking-based metric mAP@R, rather than popular Recall@K (R@K). re-evaluate 25 VL on proposed benchmarks. findings are that benchmarks, such as 1K R@K, 5K CxC R@1 highly correlated each other, while rankings change when shift ECCV mAP@R. Lastly, delve into effect bias introduced choice annotator. Source code available at https://github.com/naver-ai/eccv-caption